Collective Modes and Stability of Bose-Fermi Mixtures with a BCS-BEC Crossover 
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We investigate an ultracold Bose-Fermi mixture with a Feshbach resonance between two hyperfine 
states of fermions. Using a functional integral method, we calculate collective modes associated with 
fermion-pairs and bosons in a superfluid phase. We derive a stability condition of the mixtures, which 
is valid in the entire region of the BCS-BEC crossover. This stability condition, which smoothly 
connects the well-known results obtained in both the BCS and BEC limits, shows that sufficiently 
strong FB interactions destabilize the mixtures. In order to investigate the consequence of this 
instability, we study the ground state energy of both the uniformly mixed and the phase-separated 
states. We find that the FB repulsion induces a phase separation, whereas the FB attraction should 
cause a collapse of the mixture. The possibilities for the experimental observation of such instabilities 
are also discussed. 

PACS numbers: 03.75.Kk, 03.75.Mn 



I. INTRODUCTION 

Recent experimental studies on ultracold fermions have 
provided new opportunities to study a crossover between 
a BCS superfluid and a molecular Bose-Einstein con- 
densate (BEC). In such a system the strength of the 
two-body interaction can be freely tuned. In the weak 
two-body attraction limit, fermions form Cooper pairs 
and undergo the BCS transition. On the other hand, in 
the strong attraction limit, tightly bound pairs undergo 
Bose-Einstein condensation. Since these two limits are 
essentially understood as a condensation of the fermion 
pairs, a smooth crossover from a BCS state to a BEC 
state is expected to occur (BCS-BEC crossover). 

The study of BCS-BEC crossover was pioneered by 
Eagles [l[ and Leggett Q and later extended to finite 
temperatures by Nozieres and Schmitt-Rink Q. It was 
further extended by Sa de Melo, Randeria, and Engel- 
brecht using functional integral formalism Recent 
progress on the experimental techniques with the use 
of Feshbach resonances allows a realization of the BCS- 
BEC crossover. The crossover from BCS to tightly bound 
molecular BEC was experimentally confirmed by the ob- 
servation of pairing gap [f| , condensation fraction 0, Q , 
heat capacity p|, collective oscillations d, [nj, vortex 
lattices 11 1| and density profiles with a population im- 
balance fl2| . 

Ultracold fermion systems have been studied with 
Bose-Fermi mixtures in association with fermion cool- 
ing. Cooling of degenerate fermions was difficult since the 
inter-fermion collisions are suppressed due to the Pauli 
exclusion principle. However, by adding bosons, one can 
perform the sympathetic cooling [l3j which yields re- 
alization of ultracold fermions. In these systems, it is 
known that uniform Bose-Fermi mixtures become un- 
stable for sufficiently strong fermion-boson (FB) inter- 
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actions [3, [H, EE EE EH ; phase separation (collapse) 
occurs with repulsive (attractive) FB interactions. Ex- 
perimentally, the collapse of a Bose-Fermi mixture was 
observed in a 40 K- 87 Rb mixture Therefore, ex- 

perimental studies on ultracold fermions have been con- 
ducted to avoid the phase separation or the collapse of 
the mixture in order to perform the sympathetic cool- 
ing [2(| [Hi- Nevertheless, these instabilities themselves 
should have interesting features, thus they deserve an in- 
dependent study. 

In this paper, we investigate a Bose-Fermi mixture 
with a Feshbach resonance between two hyperfine states 
of fermions at absolute zero temperature. This system 
was recently studied by Salasnich and Toigo from the 
BCS to the unitarity region [2^]. We present a theory 
which is applicable in the whole region of the BCS-BEC 
crossover. Using a functional integral method, we cal- 
culate the collective modes associated with the quantum 
fluctuations of the boson and fermion-pair condensates. 
Assuming that the bosons and the fermions are uniformly 
mixed, we find that the Bogoliubov mode of the bosons 
and the Anderson-Bogoliubov mode of the fermions are 
mixed by the FB interactions. In order to investigate the 
system more precisely, we derive a set of coupled Gross- 
Pitaevskii equations for the boson and fermion-pair con- 
densates in the BEC limit, and show that the system is 
equivalent to two-component Bose gases. We also derive 
a stability condition against small quantum fluctuations. 
In order to understand the physical meanings of this con- 
dition, the ground state energy in the whole region of the 
crossover is derived. Using that, it is shown that the in- 
stability relates to the phase separation for repulsive FB 
interactions whereas the collapse for attractive FB inter- 
actions. Especially when the system is initially in the 
uniformly mixed state, we have shown that the instabil- 
ity against small fluctuations corresponds to the phase 
separation for FB repulsion. We also discuss how these 
instabilities can be observed in experiments. We expect 
6 Li is the best candidate to see such instabilities. 

This paper is organized as follows. In Sec. HH an 
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effective action is derived by performing a Hubbard- 
Stratonovich transformation. In Sec. IIIII an effective ac- 
tion is derived by a second order perturbation expansion 
with respect to the quantum fluctuations associated with 
the boson and fermion-pair order parameters. In Sec. lIVi 
collective modes are studied. The Bogoliubov mode of 
the bosons and the Anderson-Bogoliubov mode of the 
fermions are shown to be mixed by the FB interactions. 
In Sec. [Vj a stability condition of the mixture is derived 
and discussed. In Sec. lVIi the stability condition and the 
possibility of the experimental realization are discussed. 



II. MODEL 

We study the system at sufficiently low temperature, 
where both bosons and fermions are in a superfluid phase. 
We start with an action for a three-dimensional uniform 
Bose-Fermi mixture 

S = S B + S F + S!, (1) 

where Sb and Sf are the actions for bosons and fermions, 
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while Si represents the two-body FB interaction, 
Si[4> B ,ip^ipi\ = ^2 [ dr [ dx 

*=n Jo J 

ff/K(x,r)| 2 |0 B (x,r)| 2 (4) 

with (3 being the inverse of the temperature. Here, 
4>b{x,t) is a complex field for bosons whereas ip a (x,T) 
are Grassmann fields for fermions with spin a =T-l- 
We denote three-dimensional vectors with bold letters. 
\ib and \if are the chemical potentials for bosons and 
fermions, respectively. For simplicity, we assume that the 
numbers of the spin-up and -down particles are equal. 
We consider the boson-boson (BB) and the FB s-wave 
scatterings with coupling constants Qb = incis/mB and 
gi = 2irai/mi, respectively, clb and ai are the scatter- 
ing lengths and mi = mFiriB /{ttif + ms) is the reduced 
mass. Throughout the paper, we set h = ks = 1. 



The coupling constant for the fermion-fermion (FF) 
interaction can be controlled by a Feshbach resonance. 
The bare s-wave attraction, gp is "renormalized" as 
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where 3f = 47r aF /tuf is the effective FF interaction with 
a,F being the scattering length which can be tuned from 
— oo to +oo. 

The interaction part of Sf can be transformed using 
the usual Hubbard-Stratonovich transformation by intro- 
ducing new complex fields A(x,t) and A*(x,t). Inte- 
grating over the Grassmann fields, we obtain an effective 
action, 
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where V is the volume of the system and Q 1 is the 
inverse Nambu propagator, 
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where = (a x Ha y )/2 with <t, being the Pauli matrices. 



III. PERTURBATION EXPANSION 

Assuming that the temperature is sufficiently low, two 
order parameters, <f> B (x, r) and A(x, r), can be separated 
into their averages and fluctuations. 



4>b{x,t) = 0o + 4>{x,t), 
A(x,t) = A + S(x,t). 



(8) 
(9) 



Then, the Nambu propagator in Eq. ^ can be expanded 
as follows, 

trfln^- 1 )] 
- tr[ln(/^)]-f>[^], (10) 



where the unperturbed Green functions Qq and the self 
energy S are 

d 



1 = —aZ - ( - 
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dr V 2mF 
-A a+-A <7-, (11) 
£ = gi{<t> (<P(x,T) + <f>* {x i T)) + \<j){x,T)\ 2 )o z 

+5(x,t)<7+ +5*{x,t)(j-. (12) 
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Note that we have performed a unitary transformation 
on Q and £ to set A real. From now on we use the 
momentum representation of Qq, which are determined 
as 



-l 



A 



^ sp . ~" ) , (13) 
A iuj - £ P J 



p ' "0 

where £ p = p 2 jimp — [ip and p = (p, ioj). We now 
expand the action to the second order in the fluctuations 
around the saddle point. The saddle point conditions give 
the gap equation and the Hugenholtz-Pines relation, 

A 1 f dp 



27T) 



9F 

P-b = gsns + ginFi + giripi, 
where n B and np a are boson and fermion densities 
np 1 f dp 



(14) 
(15) 
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with np — np^ + 7ipj_ . By solving the saddle point con- 
ditions, Eqs. (fT"4l) and (fl5|) . with the particle number 
equation, Eq. (fTr!)) , self-consistently, one can calculate 
the physical quantities of this system. 

We now separate the Bose field and the pairing field 
to their real and imaginary parts 

m = -4foA(p) +»&>&>)), (17) 
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where (f> A , 4>p, S a and S p are set to be real. Before pro- 
ceeding with the expansion, we define RPA polarization 
bubbles, which emerge in the second order in the fluctu- 
ations. We define 
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Using the notations defined above, the second order ef- 
fective action is given by 
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where * p = 1/V2(0a(p) 4>p(p) S a (p) S p (p)) is a vec- 
tor notation of the fluctuation fields and /C _1 is a 4x4 
matrix 
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IV. COLLECTIVE MODES: 
ANDERSON-BOGOLIUBOV MODE AND 
BOGOLIUBOV MODE 

We now calculate the dispersion relation of the mixture 
using Eq. ([2D)) . Assuming that the frequency is linear to 
the momentum in the long-wavelength limit u> = cp, the 
RPA bubbles are expanded by u> and p to the second 
order. After performing analytical continuation iu> — > 
uj + irj, the sound velocities c are determined by solving 
det[/C _1 ] = 0. The results are plotted in Fig. Q] One 
can see that the collective excitations of the bosons and 
fermions exhibit coupled oscillations. 
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FIG. 1: Sound velocities of the Bose- Fermi mixture in unit 
of vf, the Fermi velocity in the BCS limit, calculated with 
tib/tif = 7/6, kFdi = 0.2, kpas = 0.4 and ns/riFa = 0.1 
with kF being the Fermi wave number in the BCS limit. 
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In both the BEC and the BCS limits, the dispersion 
relation is determined analytically. In the BCS limit, the 
dispersion relation reads 
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3rriB 



p\ (23) 



where vf = kp/mp is the Fermi velocity and N(0) = 
mpkp/2TT 2 is the density of states of the fermions at 
the Fermi surface with Uf = i^^np) 1 ^ ■ Two branches 
in Eq. (|23|) corresponds to the Bogoliubov mode (cb = 



V 'sb^s/tob) of the bosons and Anderson mode {ca = 
vp/V3) of the fermions in the limit gi —>■ 0; for cb > ca 
{cb < ca), the positive and negative (negative and posi- 
tive) signs correspond the Bogoliubov mode and Ander- 
son mode, respectively. 

In the BEC limit, the dispersion relation is calculated 

as 

^ = \ ^b(£b + 2g B n B ) + ec(ec + 2gcnc) 

±((es(es + 2g B n B ) - ec(ec + 2.9c«-c)) 2 
+64g 2 I e B e c n B nc) 2 ], (24) 



where e# = p 2 /2ms, ec = p 2 /Amp and np = 
A^mpap /8ir — np/2 is a fermion-pair density which will 
be discussed later. In the long-wavelength limit, Eq. 
reduces to 
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which, in the limit gi — > 0, corresponds to the Bogoliubov 
modes of the bosons and the fermion-pairs with the mass 
tylq — 2mp, inter-pair scattering length 2a p and pair 
condensate density ric — np/2. 

The result in the BEC limit can be understood more 
naturally by deriving a set of coupled Gross-Pitaevskii 
equations of the boson and fermion-pair condensates. 
Here, we employ a similar approach to the one in 
Refs. 0, HH in order to depict the tightly bound fermion 
pair. In the BEC limit, /ip is the largest scale of the 
energies since fermions are tightly bound. This enables 
us to expand Eqs. (fT4|) and (flTi)) in terms of A/\[ip\ oc 
(&F0ii?) 3//2 , which is a small parameter in the BEC limit. 
One finds [Xp and A to be 
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We now employ the off-diagonal part of the Dyson equa- 
tion of the fermion Green function by taking a variation 
SSeg/SA* = 0. We choose Go as a Green function of ideal 
fermions and the self energy part E as 

E = gi\<p(x, t)\ 2 <j z + A(x, r)a + + A*(x, r)<r - . (28) 

Then, the expansion of the Dyson equation by the self 
energy to the third order gives 
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Note that we have assumed that <7/|</>| 2 <C A and have 
neglected the second order contributions of g/|0| 2 . The 
first term in Eq. (f2"9"]l can be calculated as 
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Similarly, the remaining terms can also determined as 
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We now define a new order parameter which corresponds 
to the fermion-pair condensate density 



|*c(*,t)| 2 = ^|A ( ,,t)| 2 ~^ 
on Z 



nc- 



(33) 



Substituting Eqs. (f30lf33|) to Eq. ([29j), and using Eqs. ([26]) 
and (|2"T|) , one obtains the Gross-Pitaevskii equation of the 
fermion-pair condensates 
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where [ic — 2[i F + l/m F a F is the effective chemicai po- 
tential of the fermion-pairs. From SS g/5(ft = 0, we can 
also derive the Gross-Pitaevskii equation for the bosons, 



d ± V 2 

i— (ft 

at 2rriB 

+2 gi \^ c \ 2 <ft 



ft + g B \0\ 2 <ft 



fx B (ft. (35) 

From Eqs. (f3"4"| and (f3"5| . one can derive a set of cou- 
pled Bogoliubov de Gcnnes equations, and reproduce the 
collective modes of Eq. (|24|) . This fact implies that, 
in the BEC limit, the system is equivalent to the two- 
component Bose gases with the atomic mass of tub and 
Imp and the scattering length of as and 2a F . Note 
that the coupling constant for boson fermion-pair in- 
teraction is 2gi — Airai{mp + m,B)/m F mB rather than 
ATtai(2mp + mB)/mpmB since the FB interactions es- 
sentially act between one fermion and one boson. 



V. STABILITY CONDITION 

For sufficiently strong FB interactions, the sound ve- 
locity of the lower energy branch becomes a pure imagi- 
nary, which implies the instability of the system. Then, 
the stability boundary is determined by the condition 
that one of the sound velocities is zero. Then, the 
stability condition is expressed with the RPA bubbles, 



k(0) = fcf(0) = -fcf(0), 



as 



9b/ 9 2 i > 4/(0) + 4fc(0) 2 //(0). 



(36) 
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FIG. 2: Stability condition for Bose- Fermi mixtures plotted 
from Eq. (|36[) (solid line). The dashed and dotted line corre- 
sponds to the BCS limit, Eq. (J3TJ) and the BEC limit, Eq. (|38]l . 
respectively. 



In the BCS limit, Eq. |36]) becomes 
g B /gj > 2N(0), 



(37) 



which agrees with the thermodynamic stability condition 
for Bose-Fermi mixtures [13, EE H3, HE HE E3] • On the 
other hand, in the BEC limit, Eq. ([36]) reduces to 



g F g B > 4# 7 , 



which again coincides with the thermodynamic stabil- 
ity condition for two-component Bose gases [H, HE H3] • 
These results are summarized in Fig. [2] One can see 
Eq. (f3"rJ|) smoothly connects the well-known stability con- 
ditions of the Bose-Fermi mixtures Eq. (|3T[) and two- 
component Bose gases Eq. <f38|) . 

Till now, we have assumed that the ground state of 
the mixture is a uniformly mixed state. In order to un- 
derstand the consequence of the instability, we study 
the free energies of both the uniformly mixed states 
and the phase-separated states. At the zero tempera- 
ture, we can approximate the thermodynamic potential 
as 51 = — PV ~ /?~ 1 S'( " ) where P is the pressure, V is 
the volume and S^ 1 is the effective action without the 
quantum fluctuations. Using Eqs. (J6j) and (|10|) . and then 
setting Qq as the Green function of the ideal fermions 
and £ = gi\4>o\ 2 Cz + Aoct + + Aq<t~, the free energy of 
the uniformly mixed state is determined to be 



£mix = -p f "\n f ,v)v~p { ; c \n f ,v)v 



- flF (N F ,V)N F + ^-^ 



where 
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|A | 2 m F 

p 2 



(41) 



with Nb = n B V and N F = n F V . Note that we have 
neglected the higher order contributions of the FB inter- 
actions. 

Similarly, one can also calculate the free energy of the 
phase-separated states as 



£ S cp = -P { f °\n f ,V f )Vf - P F SC) (N F ,V F )V F 

+ f , F (N F ,V F )N F + ^-^, 
l Vb 



(42) 



with V F and Vb being the volume occupied by fermions 
and bosons, respectively. 

We now show that the stability condition Eq. (|36l) . 
relates to the condition for the system to be in a uni- 
formly mixed state rather than a phase-separated state. 
In order for phase separation to occur, the following two 
conditions must be satisfied; the equilibrium condition 
for the pressure, 



P. 



(o) 



P (SC) _ 

r J7 .-. 



9bNI 



and the condition for the free energies, 



^sep ^ii 



(43) 



(44) 



In the BEC limit, P F \ the pressure of the ideal Fermi 



(38) gas, vanishes because of the disappearance of the Fermi 
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surface. Expanding P^ sc ' by Ao/\fip\, which is a small 

parameter in the BEC limit, Pp^ — > gFfr^/2 as shown 
in Fig. [3] Remembering that the chemical potential of 
the fcrmions becomes 2\xp — > —l/mpap + gpn-c, the 
condition for phase separation becomes 

tscp tmix ~ 2 V F + 2 V B 

_9^Nb_ _5fM_ 2 NbNc 
2 V 2 V 91 V 
< 0, (45) 

where Nc = Np/2 is a fermion pair number. Using 
the equilibrium condition Eq. (|43p . which reduces to 
gpNQ/2Vp — gsNg/V^, the condition for phase sep- 
aration in the BEC limit becomes 

45? > 9B9F (46) 

for repulsive FB interactions. Eq. (|46|) agrees with the 
condition for phase separation of the two-component 
Bose gases [2{| [3l|. Phase separation of the two- 
component Bose gases was actually observed in the ex- 
periment [32j . This condition also agrees with the stabil- 
ity condition against the small fluctuations in the BEC 
limit, Eq. pgjl. 




-4-2 2 4 
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FIG. 3: Fermi pressure Pp + Pp plotted against interac- 
tion parameter (solid line). In the BCS limit, it approaches 
to 2/5/zfWf (gray dashed line) whereas in the BEC limit, it 
approaches gFnc/2 (gray dotted line). 

In the BCS limit, Pp^ is negligible compared to P F °^ 
since A /ep <C 1. Then the argument on the Bose- Fermi 
mixture with one-component fermions also applies to our 
case. Using the results of the previous work 
which basically compares the ground state energies of the 
uniformly mixed states and the phase-separated states, 
Eq. (j3"Tj) corresponds to the condition that the system 
does not have any phase-separated states in the dilute 
boson density limit. 

Therefore, with repulsive FB interactions, Eq. (f3"o| ex- 
actly corresponds to the condition for phase separation 
to occur in the BEC limit, whereas it corresponds to 



the condition only in the limiting case in the BCS limit. 
However, in the next section, we will show that the phase 
diagram is determined by Eq. (|36|) regardless of the den- 
sity of bosons if the system is initially in a uniformly 
mixed state. 

We point out that Salasnich and Toigo also found the 
stability conditions based on the approximation of the 
ground state energy calculated by a Monte Carlo study 
[221 ]. They explicitly calculated the stability conditions 
which agree with our findings in the BCS limit. However, 
their approach is currently limited in the BCS to the 
unitarity region. In contrast, our theory is also valid in 
the BEC region. 

When the FB interactions are attractive, the condition 
Eq. (|44f is not satisfied since an increase of \gi\ lowers the 
ground state energy of the uniformly mixed state. There- 
fore, the phase-separated state is unstable. In this case, 
we expect the mixture to collapse by the FB attraction. 



VI. DISCUSSION 

In this section, we discuss how our stability condi- 
tion can be verified in real systems. We point out that 
the phase diagram is governed by the stability condition 
Eq. (|36p assuming that the system is initially in the uni- 
formly mixed state. 

The stability condition is derived so that the system is 
robust against the fluctuations. In other words, it is the 
condition that a uniformly mixed ground state is at the 
local minimum of the free energy. Consider the case that 
the interaction parameter is swept adiabatically from the 
BCS to the BEC region. At sufficiently low temperatures, 
there is no process for the system to make a transition 
to the other state. Therefore, the system remains at its 
initial state, even when a non uniformly-mixed state, e.g. 
the phase-separated state, is energetically favored due 
to the confining potential. However, once the condition 
Eq. (|3"6"|) is violated, the system becomes unstable against 
any small fluctuations. Therefore, its phase diagram is 
expected to be like Fig. [2] when the system is initially in 
a uniformly mixed state. 

This explains the contradiction between our results 
and that of the experiment of the 40 K- 87 Rb mixture [lj|. 
In this experiment, the condition for the collapse depends 
on the densities of both fermions and bosons whereas 
Eq. (|36[) only depends on the fermion density. This can 
be explained by considering that the initial state of the 
experiment where the collapse was observed was not a 
uniformly mixed state. In this case, the stability condi- 
tion should be governed by £ ther > £mix where father is 
the ground state energy of the state after the collapse. 

We now discuss how one could experimentally demon- 
strate our results. In order to demonstrate the stability 
condition Eq. (|36[> . we first need to prepare a uniformly 
mixed state and then change some parameters to make 
the system unstable. Presently, it seems difficult to con- 
trol the FB interactions. However, one can still tune 
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dp to make the system unstable. Hence, if a uniformly 
mixed state was successfully realized in the BCS or in the 
unitarity region, where the system is likely to be stable 
(see Fig. [3]) , the transition to the other ground state can 
be observed by tuning ap to the BEC region. 

Currently there are two types of systems in which the 
BCS-BEC crossover was observed % i, 0, 1, 1 E3 HO, 
03; 40 K of the hyperfine states of \f,m f ) = |9/2, —7/2) 
and |9/2, —9/2) and 6 Li of |l/2, 1/2) and |l/2, —1/2). 
As far as we know, there has been no FB interaction 
scattering length reported in these states. However, we 
can still estimate the magnetic field strength where the 
instability occurs using an approximate expression of a^, 

a,~a bg (l-^), (47) 

where B is the magnetic field strength, Bq is the magnetic 
field at the resonance, a^g is the scattering length in the 
small or large B limit and \AB\ is corresponded to the 
width of the resonance. 

In the case of 40 K, a^g = 174ao [34[ where a is Bohr 
radius. Taking ab g as the scattering length in the BEC 
limit, the range of the parameters where instability can 
be observed is 174ao < (l+m_B) 2 aj/(4msa_B) with raj = 

The observation of the transition is more promising in 
the case of 6 Li since a^g = — 1405ao [HI]. The system 
is in the pure BEC limit, i.e. aj? = +0 at B = 534G. 
Here, Eq. (|36|) automatically becomes false. Therefore, if 
one could realize a uniformly mixed state in the BCS or 
the unitarity region and tune the Of to the BEC region, 
one should see the instability for B > 543G. However, 
the experiment shows that the fermion-pair condensate 
vanishes around B ~ 710G Therefore, in order to see 
the transition such as phase separation and collapse, an 
appropriate hyperfine state which fails to meet Eq. (I36|) 
in B > 710G must be chosen. 

In experiments, mixtures are trapped by an external 
trap potential, which is not considered in our present 
homogeneous theory. Assuming Nb and Np ^> 1, 
the effect of a trap can be reasonably investigated 
by performing a local density approximation, which 
modifies fip and fiB to [ip.jiB — > MFjMb — V ex t(x) 
where V ex t(x) is the external trap potential. Normal- 
ization conditions, e.g. Eq. (|16p . are also changed 
to N F = ft- 1 J dxY, p Go(p;p>F-Vext(x)) and N B 
| dx riB (x; \ib — Kxt (x)). This approximation fails to 
describe the situation at the edge of the trap. This can 
be understood by considering the case that the same 
number of fermions and bosons are present in the same 
trap, and there is no FB interaction. Due to the Pauli 
exclusion principle, fermions are usually spread more 
than bosons, therefore, even though no FB interaction 
is present, phase separation can take place at the edge of 
the trap. 

However, this approximation well describes the physics 
at the center of the trap, where instability due to fluctu- 
ations is expected to occur first. The stability condition 



Eq. (|36|) . which is found for a homogeneous case, is equiv- 
alent to the stability condition at the center of the trap 
which is described within a local density approximation. 
Therefore, we expect that our condition is still useful for 
trapped systems. However, it should be pointed out that 
for FB repulsion, the effect of instability may be less vis- 
ible for the trapped case since the instability occurs only 
at the center of the trap. On the contrary, instability oc- 
curs simultaneously in the all region for the homogeneous 
case. For the FB attraction, instability at the center of 
a trap induces instability in the all region, therefore, ex- 
perimentally, the phase diagram Fig. [5] could be observed 
more clearly. 

VII. CONCLUSION 

We have studied a system of a Bose-Fermi mixture 
with a Feshbach resonance between two hyperfine states 
of fermions in a superfluid phase. It has been shown 
that the Anderson-Bogoliubov mode and the Bogoliubov 
mode are mixed by FB interactions. We have also shown 
that the system becomes unstable as the strength of the 
FB interactions increases. The stability condition de- 
rived here has been shown to smoothly connect the well- 
known results of both the BCS and BEC limits. In order 
to understand the stability condition, we have studied 
the ground state energy of this system for both the uni- 
formly mixed and the phase-separated states. It has been 
shown that the stability condition relates to the condition 
for phase separation if the FB interactions are repulsive. 
Especially when the system is initially in the uniformly 
mixed state, we have shown that it corresponds to the 
condition for phase separation. We have also shown that 
if a uniformly mixed Bosc-Fcrmi mixture was prepared 
experimentally, the stability condition Eq. (|36[) can be 
verified. We expect 6 Li is the best candidate for the 
transition to phase separation or collapse to be observed. 
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